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The  NATO  Science  and  Technology  Organization 


Science  &  Technology  (S&T)  in  the  NATO  context  is  defined  as  the  selective  and  rigorous  generation  and  application  of 
state-of-the-art,  validated  knowledge  for  defence  and  security  purposes.  S&T  activities  embrace  scientific  research, 
technology  development,  transition,  application  and  field-testing,  experimentation  and  a  range  of  related  scientific 
activities  that  include  systems  engineering,  operational  research  and  analysis,  synthesis,  integration  and  validation  of 
knowledge  derived  through  the  scientific  method. 

In  NATO,  S&T  is  addressed  using  different  business  models,  namely  a  collaborative  business  model  where  NATO 
provides  a  forum  where  NATO  Nations  and  partner  Nations  elect  to  use  their  national  resources  to  define,  conduct  and 
promote  cooperative  research  and  information  exchange,  and  secondly  an  in-house  delivery  business  model  where  S&T 
activities  are  conducted  in  a  NATO  dedicated  executive  body,  having  its  own  personnel,  capabilities  and  infrastructure. 

The  mission  of  the  NATO  Science  &  Technology  Organization  (STO)  is  to  help  position  the  Nations’  and  NATO’s  S&T 
investments  as  a  strategic  enabler  of  the  knowledge  and  technology  advantage  for  the  defence  and  security  posture  of 
NATO  Nations  and  partner  Nations,  by  conducting  and  promoting  S&T  activities  that  augment  and  leverage  the 
capabilities  and  programmes  of  the  Alliance,  of  the  NATO  Nations  and  the  partner  Nations,  in  support  of  NATO’s 
objectives,  and  contributing  to  NATO’s  ability  to  enable  and  influence  security  and  defence  related  capability 
development  and  threat  mitigation  in  NATO  Nations  and  partner  Nations,  in  accordance  with  NATO  policies. 


The  total  spectrum  of  this  collaborative  effort  is  addressed  by  six  Technical  Panels  who  manage  a  wide  range  of 
scientific  research  activities,  a  Group  specialising  in  modelling  and  simulation,  plus  a  Committee  dedicated  to 
supporting  the  information  management  needs  of  the  organization. 
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well  as  recognised  world-class  scientists,  engineers  and  information  specialists.  In  addition  to  providing  critical 
technical  oversight,  they  also  provide  a  communication  link  to  military  users  and  other  NATO  bodies. 


The  scientific  and  technological  work  is  carried  out  by  Technical  Teams,  created  under  one  or  more  of  these  eight 
bodies,  for  specific  research  activities  which  have  a  defined  duration.  These  research  activities  can  take  a  variety  of 
forms,  including  Task  Groups,  Workshops,  Symposia,  Specialists’  Meetings,  Lecture  Series  and  Technical  Courses. 
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An  Applied-Scientific  Approach  to  Field  Assessments  - 
“Try  It  Out”  -  A  Field  Guide  for  Practitioners 

(STO-TR-HFM-211) 

Executive  Summary 

In  NATO  and  in  many  nations  a  wide  range  of  innovations  and  improvements  in  technology,  information 
processes  and  organisational  structures  are  being  developed  and  tried  out  by  the  military  in  the  context  of 
new  operational  settings  and  requirements.  When  it  comes  to  deciding  whether  an  innovation  or  intended 
improvement  actually  adds  to  the  operational  performance  this  assessment  guide  is  intended  to  support 
practitioners,  in  this  case  commanders  and  staff. 

Rather  than  going  for  high  level  scientific  rigor  this  guide  aims  to  provide  a  practical  method  to  improve 
assessment  efforts  that  might  be  based  on  simple  methods  of  just  observing  if  it  works.  Investing  in  simple 
measures  to  improve  the  assessment  practise  may  reduce  the  number  and  magnitude  of  potential  errors  and 
misleading  conclusions.  At  the  same  time  such  efforts  will  increase  the  value  of  the  insights  coming  from 
experimentation  in  natural  settings  -  be  it  field  trials,  field  experiments,  tests  or  evaluations. 

This  guide  is  based  on  experience  of  the  the  NATO  STO  Task  Group  (HFM-21 1)  with  assessment  in  practice 
and  is  based  on  knowledge  of  scientific  rigour.  Starting  from  the  premise  that  there  is  an  identified  need 
coming  from  the  commander  or  staff  or  other  stakeholder  for  an  assessment  to  support  a  grounded  decision, 
the  Guide  takes  its  user  in  five  steps  from  problem  exploration  to  planning  and  execution  to  reporting: 

•  Framing  the  Question  -  In  dialogue  with  the  problem  owner,  often  the  commander,  the  basic 
aspects  of  the  question  the  underlying  problem  space  and  available  resources  are  identified. 

•  Initial  Problem  Space  Analysis  -  Based  on  a  deeper  analysis  of  the  question  and  the  problem  space 
a  brief  initial  planning  of  the  assessment  task  is  developed  by  the  assessment  team.  This  should 
allow  the  commander  to  decide  on  the  size  of  the  assessment  and  the  available  resources. 

•  Detailed  Assessment  Planning  -  The  assessment  team  conducts  an  in  depth  planning  of  the 
assessment  task.  This  includes  development  of  a  detailed  assessment  plan  with  identified  indicators 
and  corresponding  procedures  to  collect  information. 

•  Preparation  and  Execution  -  The  assessment  team  manages  the  practical  preparation,  setup  of  the 
experimentation,  and  conducts  the  data  collection. 

•  Analysis  and  Report  -  The  analysis  is  directed  to  answering  the  formulated  assessment  question. 
Also  an  assessment  must  be  made  of  the  quality  of  the  obtained  data,  in  order  to  value  the  results. 

The  five  steps  are  supported  by  a  set  of  twenty-two  leading  questions  that  are  supposed  to  trigger  the 
considerations  to  be  addressed  in  the  assessment  process.  The  Guide  also  takes  into  account  that  time  and 
resources  of  the  staff  are  limited  due  to  primary  operational  tasks  and,  moreover,  that  no  scientific  support  is 
available  to  support  the  assessment.  Modelled  against  the  generic  command  process,  familiar  to  the  intended 
practitioners,  the  assessment  process  can  be  incorporated  in  the  staffs  daily  schedule  and  mimic  the  quality 
control  steps  from  the  command  process.  While  this  Guide  is  based  on  our  practical  experiences  it  has  not 
been  tested  or  validated  in  practice  by  the  Task  Group.  Feedback  from  defence  colleges  and  Training  and 
Evaluation  staffs  is  highly  welcomed. 
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Une  approche  scientifique  des  evaluations  de  terrain  - 
«  Essayer  le  »  -  Un  guide  de  terrain  a 
destination  des  praticiens 

(STO-TR-HFM-211) 

Synthese 

Les  militaires  de  l’OTAN  et  de  nombreux  pays  elaborent  et  essaient  une  large  gamme  d’innovations  et 
d’ ameliorations  en  matiere  de  technologie,  precedes  d’information  et  structures  organisationnelles  pour  faire 
face  a  de  nouveaux  parametres  et  de  nouvelles  exigences  operationnelles.  Le  present  guide  devaluation  est 
conqu  pour  aider  les  praticiens,  en  l’occurrence  les  commandants  et  le  personnel,  a  decider  si  une  innovation 
ou  une  amelioration  prevue  renforce  reellement  les  performances  operationnelles. 

Au  lieu  de  preconiser  une  rigueur  scientifique  extreme,  ce  guide  propose  une  methode  pratique  qui  pourrait 
reposer  sur  de  simples  observations  des  resultats.  L’adoption  de  mesures  simples  ameliorant  la  pratique 
devaluation  est  susceptible  de  reduire  le  nombre  et  l’etendue  des  erreurs  potentielles  et  des  conclusions 
trompeuses.  Elle  augmentera  la  valeur  des  informations  issues  de  1’ experimentation  dans  un  cadre  naturel, 
qu’il  s’agisse  d’essais  ou  d’experiences  sur  le  terrain,  d’essais  de  laboratoire  ou  devaluations. 

Ce  guide  s’appuie  sur  l’experience  du  groupe  de  travail  HFM-211  de  la  STO  de  LOT  AN  en  matiere 
devaluation  pratique  et  sur  la  rigueur  scientifique.  Partant  du  principe  que  le  commandant,  le  personnel  ou 
un  autre  acteur  a  besoin  d’une  evaluation  pour  prendre  une  decision  justifiee,  le  guide  presente  cinq  etapes 
allant  de  f  etude  du  probleme  jusqu’au  compte  rendu,  en  passant  par  la  planification  et  l’execution  : 

•  Cerner  la  question  -  Les  aspects  fondamentaux  de  la  question,  le  domaine  sous-jacent  du  probleme 
et  les  ressources  disponibles  sont  identifies  en  collaboration  avec  la  personne  chargee  de  resoudre  le 
probleme,  frequemment  le  commandant. 

•  Analyse  initiale  du  domaine  du  probleme  -  A  partir  d’une  analyse  approfondie  de  la  question  et 
du  domaine  du  probleme,  l’equipe  devaluation  elabore  un  plan  sommaire  initial  de  la  tache 
devaluation,  ce  qui  devrait  permettre  au  commandant  de  decider  de  l’ampleur  de  1’evaluation  et  des 
ressources  disponibles. 

•  Planification  detaillee  de  1’evaluation  -  L’equipe  devaluation  realise  une  planification 
approfondie  de  la  tache  devaluation.  11  s’agit  notamment  d’etablir  un  plan  devaluation  detaille  avec 
des  indicateurs  identifies  et  des  procedures  de  recueil  d’information. 

•  Preparation  et  execution  -  L’equipe  devaluation  gere  la  preparation  pratique  et  la  mise  en  place 
de  1’  experimentation  et  dirige  le  recueil  des  donnees. 

•  Analyse  et  compte  rendu  -  L’analyse  est  orientee  de  maniere  a  repondre  a  la  question  devaluation 
formulee.  11  est  egalement  imperatif  d’evaluer  la  qualite  des  donnees  obtenues,  afm  d’evaluer  les 
resultats. 

Ces  cinq  etapes  prennent  appui  sur  un  jeu  de  vingt-deux  questions  principales  censees  indiquer  les  points  a 
traiter  au  cours  du  processus  devaluation.  Le  guide  tient  compte  du  fait  que  le  temps  et  les  ressources  du 
personnel  sont  limites  par  des  taches  operationnelles  primaires  et  que,  par  ailleurs,  aucun  soutien  scientifique 
n’est  mis  a  disposition  lors  de  1’evaluation.  Etabli  d’apres  le  processus  de  commandement  general,  auquel  les 
praticiens  concemes  sont  habitues,  le  processus  d’ evaluation  peut  etre  integre  dans  le  programme  quotidien 
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du  personnel  et  iinite  les  etapes  de  controle  de  la  qualite  du  processus  de  commandement.  Ce  guide  se  fonde 
sur  nos  experiences  pratiques,  mais  n’a  pas  ete  teste  ni  valide  en  pratique  par  le  groupe  de  travail. 
Nous  apprecierions  un  retour  d’experience  des  colleges  de  la  defense  et  des  equipes  de  formation  et 
d’ evaluation  de  1’ instruction. 
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AN  APPLIED-SCIENTIFIC  APPROACH  TO  FIELD  ASSESSMENTS  - 
“TRY  IT  OUT”  -  A  FIELD  GUIDE  FOR  PRACTITIONERS 

1.0  INTRODUCTION 
1.1  Background 

In  NATO  and  in  many  nations  a  wide  range  of  innovations  and  improvements  in  technology,  information 
processes  and  organisational  structures  are  being  developed  and  tried  out  by  the  military  in  the  context  of  new 
operational  settings  and  requirements.  Experimentation  and  evaluation  programs  have  been  setup  to  analyse  and 
demonstrate  the  added  value  and  effectiveness  of  new  developments  such  as  new  operational  concepts,  new 
information  systems,  new  organisational  structures,  etc. 

Assessment  of  effectiveness  of  innovations  and  intended  improvements  of  operational  functioning  has  a  long 
history  with  many  systematic  approaches,  such  as,  e.g.  The  Army  Command  and  Control  Evaluation  System  - 
ACCES  [1],  and  U.S.  Navy  Headquarters  Effectiveness  Analysis  Tool  (HEAT)  [2],  In  addition,  different  Codes 
of  Best  Practice  (such  as  GUIDEx  [3],  NATO  Code  of  Best  Practice  of  C2  Assessment  [4])  and  reviews  [5]  lay 
down  the  basic  principles  of  assessments  conducted  by  teams  of  qualified  scientists  and  their  staff.  Existing 
approaches  and  codes  provide  adequate  guidance  and  sources  of  information,  but  at  the  same  time  they  often 
demand  or  assume  a  high  level  of  scientific  control,  which  is  difficult  to  obtain  in  military  practice.  Still  driven 
by  operational  needs  and  perceived  problems  of  effectiveness  operational  practice  constantly  performs 
exploration  of  and  experimentation  with,  amongst  others,  new  technologies,  support  systems,  new  ways  of 
working,  new  organizational  structures  for  current  or  changed  conditions.  Doing  this  light-heartedly  might  lead 
to  missed  opportunities  or  just  false  conclusions  and  hindsight  costs. 

As  we  have  observed  over  the  years  while  working  with  the  military  there  seems  to  be  a  gap  between  the 
operational  practice  of  assessment  and  the  scientific  rigor  that  is  being  advocated  in  the  documents  mentioned 
above.  In  practice  experimenting  with,  e.g.  new  tools  or  ways  of  working  is  often  based  on  informal  methods  to 
get  an  impression  of  the  added  value  of  the  new  thing  and  deciding  on  that  basis.  At  the  same  time  we  have  seen 
that  there  is  an  appreciation  and  a  need  for  more  solid  methods  if  these  would  fit  the  always  limited  time  and 
resources  of  operational  practice.  To  respond  to  this  need  we  have  set  up  a  Task  Group  (HFM-211)  to  develop 
an  approach  for  field  assessments,  maintaining  that  scientific  rigor  is  beyond  the  possibilities  of  operational 
practice  and  therefore  starting  from  the  practice  side  exploring  how  to  improve  the  quality  of  assessment  within 
the  reach  of  operational  practice.  Investing  in  the  quality  of  the  operational  practice  of  assessment  might  reduce 
potential  errors  of  misjudging  the  effectiveness  of  certain  developments  and  at  the  same  time  increase  the  value 
of  the  insights  coming  from  such  experimentation. 

The  guide  is  based  on  a  problem  analysis  approach  supported  by  a  set  of  leading  questions  to  trigger  and  focus 
the  thinking  about  the  assessment.  The  questions  represent  important  considerations  to  be  addressed  during  the 
planning,  execution  and  evaluation  of  an  assessment  task.  The  term  ‘assessment’  is  being  used  differently  in 
different  contexts.  Here  assessment  is  the  process  of  making  a  measurement  and  judgment  of  the  value  or  worth 
of  an  entity  or  new  situation  (e.g.  a  technology,  process,  program,  procedure)  in  the  context  of  the  functioning  of 
a  system,  group,  unit).  An  essential  element  of  the  assessment  is  creating  a  set-up  for  the  assessment  as  a  trial  or 
experiment  in  such  a  way  that  conclusions  can  be  drawn  about  the  problem  stated  or  question  asked.  In  military 
context  assessment  is  also  used  in  the  context  of  ‘operations  assessment’  which  is  about  “continually  monitor  the 
Operational  Environment  (OE)  and  assess  the  progress  of  the  operation  toward  the  desired  end  state”  [6].  While 
sometimes  used  interchangeably,  the  label  ‘evaluation’  is  mostly  being  given  to  summative  assessments  to 
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determine  the  relevance  and  fulfilment  of  objectives,  development  efficiency,  effectiveness,  impact  and 
sustainability.  Evaluation  practice  provides  useful  ideas  and  methods  for  assessment  (see  Refs.  [7]  and  [8]). 

The  objective  of  this  report  is  to  compile  a  practical  approach  to  assessment  in  natural  settings,  such  as  field 
trials,  field  experiments,  tests  or  evaluations,  in  the  form  of  a  guide  for  practitioners  -  commanders  and  tasked 
personnel. 

1.2  Approach 

The  Task  Group  was  preceded  by  an  exploratory  team  that  organised  a  workshop  in  Soesterberg,  Netherlands, 
March  2010,  to  explore  the  issues  of  C2  functioning  assessments  in  an  operational  context.  In  a  supporting  TNO- 
report  “Elements  of  a  Field  Guide  for  Practitioners”  (TNO-DV  2011  IN017)  possible  elements  of  such  a  guide 
were  listed  and  reviewed.  It  was  concluded  that  central  to  the  Field  Guide  should  be  the  concept  of  ‘dialogue’ 
between  the  operator  and  researcher.  Questions  were  drawn  up  in  the  Socratic  tradition.  The  overall  purpose  of 
these  questions  is  to  challenge  accuracy  and  completeness  of  thinking  in  a  way  that  acts  to  move  people  towards 
their  ultimate  goal.  The  proposal  was  that  such  questions  approach  should  be  the  core  of  the  Field  Guide. 
The  Exploratory  team  produced  the  Terms  of  Reference  for  the  Task  Group. 

The  Task  Group  gathered  in  seven  meetings  from  November  2010  to  November  2013,  and  invited  external 
experts  to  discuss  the  intended  product.  In  the  first  meetings  all  elements  of  formal  and  informal  assessment 
were  explored.  In  the  final  four  meetings  convergence  of  all  aspects  was  aimed  for  in  a  format  that  would  suit 
practitioners.  This  resulted  in  a  basic  structure  and  content  of  a  draft  report. 

Reflection  on  this  result  suggested  that  it  was  still  too  academic  and  less  fitting  the  intended  practitioners  view. 
This  required  some  rethinking  of  the  approach.  The  main  step  to  the  current  version  of  the  Field  guide  was  to 
link  the  assessment  process  to  the  command  process  as  an  analogous  process.  The  assumption  was  that  linking  to 
a  natural  way  of  practitioners’  thinking  of  clear  progressing  steps  and  quality  control  elements  with  a 
commander  (or  other  leader)  as  the  problem  owner  and  decision  maker  concerning  direction  and  extends  of  the 
assessment. 

The  Guide  is  based  on  scientific  and  operational  experiences  and  insights  of  the  Task  Group  participants,  but 
must  be  seen  as  a  Concept  document.  No  validation  in  military  settings  has  been  done  so  far.  The  Guide  is  open 
for  further  development  and  ‘trying  it  out’. 

1.3  Positioning  of  the  Guide 

Scientific  approaches  to  assessment  seek  high  quality  of  assessments,  while  the  practical  context  of  assessments 
can  be  characterized  as  having  little  time  and  resources  available  for  performing  assessments.  Quality  in  the 
latter  assessments  may  be  suboptimal,  with  the  risk  of  accepting  false  positives  or  rejecting  an  opportunity. 
Operators  could  benefit  to  achieve  methodologically  stronger  results  by  increasing  the  quality  of  their 
assessments. 

Taking  quality  of  assessment  as  one  dimension  and  size  of  resources  as  the  other  our  perspective  is  that  higher 
levels  of  quality  can  primarily  be  assured  by  involving  individuals  with  training  in  the  scientific  method. 
However,  for  practical  reasons  it  is  not  possible  or  feasible  to  have  scientific  support  in  all  assessment  tasks. 
Depending  on  the  problem  to  solve,  the  need  of  resources  varies.  The  intended  use  and  audience  of  the  Guide, 
we  envisaged,  are  operational  and  tactical  levels  commanders  and  staff  and  a  situation  in  which  there  is  no 
support  from  science  to  address  a  problem  by  a  dedicated  assessment  process.  The  guide  is  intended  to  work  at 
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the  practitioner’s  side  and  provide  ways  to  improve  the  quality  of  assessment  while  staying  within  the 
possibilities  of  practical  conditions.  This  leads  to  four  quadrants  with  is  illustrated  in  the  figure  below  (Figure  1). 
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Quality 


Low 


'Ideal' 
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Scientific  support 
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Guide  Focus 
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Own  Staff 

Own  staff 
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expertise 

expertise 

Low  Resources  High 

Figure  1 :  The  Scope  of  the  Assessment  Guide. 

Thus,  given  practice  limitations  and  limited  resources  one  would  end  up  in  the  lower  left  quadrant.  The  Guide 
focusses  on  that  quadrant  aiming  to  improve  the  quality  of  those  assessments. 

The  following  conditions  we  assume: 

•  A  need  for  an  assessment  to  support  a  grounded  decision  has  emerged  for  instance  resulting  for  a 
perceived  problem  to  be  addressed,  or  need  to  explore  the  optimal  implementation  of  new  equipment  or 
new  process  and  procedure. 

•  Time  and  resources  of  the  staff  are  limited  due  to  primary,  operational,  tasks.  Thus  the  commander  has 
to  make  a  decision  on  if  (is  it  worthwhile)  the  assessment  should  be  done  and  what  resources  are 
available  for  that. 

•  No  scientific  support  is  available  to  support  the  assessment. 

•  Assessment  is  performed  by  an  empirical  approach,  be  it  straightforward  trying  out  or  ‘experimenting’ 
or  testing  with  few  or  many  participants. 

Given  this  context  of  limited  possibilities  the  guide  should  provide  the  commander  and  the  tasked  personnel  a 
relatively  simple  and  understandable  procedure  (therefore  no  templates)  to  add  quality  to  the  assessment.  Quality 
principles  may  relate  to,  for  instance,  understanding  the  problem,  formulating  alternative  hypotheses, 
triangulation  of  methods  (using  multiple  methods). 
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For  the  purpose  of  this  guide,  we  use  the  term  commander  to  indicate  the  individual  who  has  the  mandate  to 
make  the  adequate  decisions  to  bring  the  process  forward  and  has  the  responsibility  to  initiate  an  assessment. 
We  assume  in  this  guide  that  the  commander  has  a  stake  in  the  assessment  even  if  it  is  not  the  commander  who 
is  the  “owner  of  the  problem.”  In  addition,  the  commander  has  the  final  say  on  what  resources  are  available  for 
the  assessment.  Therefore,  the  assessment  process  is  designed  around  successive  decisions  to  be  made  by  the 
commander.  In  practice,  it  might  be  necessary  to  get  approval  from  other  stakeholders  -  “problem  owners”  - 
as  well. 

1.4  The  Assessment  Process  Adapted  to  the  Staff  Process 

A  generic  command  and  control/staff  process  is  used  as  the  basis  for  outlining  an  assessment  process.  It  is 
important  to  note  that  the  process  is  generic  and  conceptual.  Specific  processes  for  tactical  and  operational 
command  may  vary  between  countries  and  units.  In  addition,  terms  describing  different  sub-  processes  may  also 
vary.  For  the  purpose  of  this  guide,  the  generic  model  below  is  assumed  to  capture  the  essence  (Figure  2). 
In  practice,  one  might  need  to  adjust  to  the  command  process  that  is  being  used  in  the  unit  involved. 
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Figure  2:  The  Generic  Command  and  Control  (Staff) 
Process  and  the  Analogues  Assessment  Process. 


The  command  process  is  assumed  to  start  and  end  with  the  commander;  the  staff  provides  the  information  and 
options  for  the  commander  to  decide  upon.  This  staff  work  comprises  the  detailing  and  extending  of  the  tasking, 
gathering  the  relevant  information  and  organising  the  processes  to  provide  that  information.  Regular  briefs  are 
undertaken  in  which  the  commander  is  given  adequate  and  sufficient  information  to  make  decisions  on  how  to 
proceed. 

The  process  is  initiated  when  an  order  or  a  task  is  received  by  the  unit.  An  initial  task/mission  analysis  is 
performed  which  results  in  a  stated  “commanders  (initial)  intent”.  Based  on  “commander’s  intent”  and 
contextual  factors  such  as  the  enemy  and  the  weather  an  initial  planning  is  executed  in  which  different  options  - 
Courses  Of  Actions  (CO A)  -  to  solve  the  task  are  developed  and  compared.  After  the  initial  planning  brief, 
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the  commander  takes  a  principal  decision  on  how  to  execute  the  task  -  to  approve  one  of  the  COAs.  This  is  the 
foundation  for  distributing  warning  orders  and  the  subsequent  detailed  planning.  The  detailed  planning  is  about 
the  successive  development  of  an  order  -  for  example  an  OPLAN  or  OPORD. 

The  result  of  the  detailed  planning  is  presented  in  a  final  planning  brief1.  Based  on  this,  the  commander  makes 
the  formal  decision  to  execute  the  task  according  to  the  order  outlined  in  the  planning  process.  The  execution  is 
then  managed  by  the  operations  section  of  the  staff.  Regular  update  briefs  are  given  in  order  for  the  commander 
to  make  necessary  decisions  based  on  how  the  situation  unfolds.  When  the  task  is  completed,  a  task/operation 
evaluation  should  be  undertaken  and  summarized  in  lesson  learned  briefs/documentation.  Based  on  identified 
lessons  learned,  the  commander  makes  necessary  decisions  to  address  the  issues  identified  in  lessons  learned. 

The  corresponding  assessment  process  outlined  in  this  guide  is  modelled  on  the  command  process  described 
above.  Consequently  it  includes  similar  successive  steps  each  concluded  by  a  decision  on  how  to  proceed. 
An  overview  of  the  two  processes  is  shown  in  Figure  2. 

1.5  Outline  of  the  Guide 

The  five  steps  identified  are  summarized  below  and  are  described  in  more  detail  in  the  Guide,  which  is  presented 
in  Section  2.0  of  this  report  (a  short  hand-out  overview  of  the  whole  process  -  the  five  steps  and  the  twenty-two 
leading  questions  -  can  be  found  in  Figure  4): 

Step  1)  Framing  the  Question  -  In  dialogue  with  the  owner  of  the  problem,  in  this  guide  the  commander, 
the  basic  aspects  of  the  question  and  the  underlying  problem  space  and  available  resources  for 
initiating  the  next  step  in  the  work  is  clarified.  The  first  section  concludes  with  a  decision  by  the 
commander  whether  to  make  an  initial  analysis  of  the  problem.  The  decision  should  include  a 
clearly  defined  “commander’s  problem  statement”  and  which  resources  are  available  for  initial 
planning.  The  process  in  detail  is  outlined  in  Section  2.1  of  this  report. 

Step  2)  Initial  Problem  Analysis  -  An  assessment  team  executes  a  brief  initial  planning  of  the  assessment 
task.  The  planning  team  reviews  and  develops  the  “commander’s  problem  statement”.  The  result  of 
this  work  is  presented  in  a  “planning  brief’.  The  aim  is  to  be  able  to  decide  whether  the  assessment 
is  worth  the  effort.  Based  on  the  information  given  at  the  brief,  the  commander  decides  on  how  to 
proceed.  The  process  in  detail  is  outlined  in  Section  2.2  of  this  report. 

Step  3)  Detailed  Assessment  Planning  -  The  assessment  team,  eventually  expanded  with  additional 
resources,  then  makes  an  in  depth  planning  of  the  assessment  task.  This  includes  development  of  a 
detailed  assessment  plan  including  definitions  of  indicators  and  corresponding  procedures  to  collect 
information  about  these  indicators.  The  result  from  the  assessment  planning  is  presented  in  an 
“assessment  brief’.  Based  on  the  information  given  at  the  brief,  the  commander  decides  on  whether 
to  proceed  or  not  with  the  assessment.  The  process  in  detail  is  outlined  in  Section  2.3  of  this  report. 

Step  4)  Preparation  and  Execution  -  The  assessment  team  then  manages  the  practical  preparation  and 
execution  of  the  assessment,  e.g.  collection  of  information/data  about  the  identified  indicators, 
with  their  assigned  resources.  The  phase  results  in  a  compiled  set  of  data/information  about  the 
stated  problem.  The  results  are  presented  in  a  first  impression  brief.  The  process  in  detail  is  outlined 
in  Section  2.4  of  this  report. 

Step  5)  Analysis  and  Reporting  -  The  assessment  team  analyses  the  compiled  data.  The  approach  for  how 
to  do  this  is  highly  dependent  on  the  specific  contextual  circumstances  and  should  be  defined  in  the 


During  the  detailed  planning,  regular  update  briefs  on  how  the  planning  proceeds  are  normally  given. 
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assessment  plan.  The  main  expected  outcome  is  of  course  an  answer  on  the  stated  problem. 
In  addition,  the  team  should  also  make  an  assessment  on  the  quality  of  the  obtained  data.  The  result 
is  presented  in  a  “results  brief’.  Based  on  the  information  given  at  the  brief,  the  commander  decides 
on  any  further  action.  The  process  in  detail  is  outlined  in  Section  2.5  of  this  report. 

The  steps  above  can  also  be  viewed  from  a  resource  perspective.  Successively  more  and  more  resources  are  used 
for  each  step  up  until  the  preparation  and  execution  step  which  presumably  is  the  most  resource  intensive  phase. 
The  subsequent  analysis  and  report  phase  is  less  personnel  intensive.  Of  course,  resources  needed  for  the  final 
step,  actions  to  implement  lessons  learned,  varies  depending  on  the  addressed  problem.  These  resource 
considerations  are  illustrated  in  Figure  3. 
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Perceived  Decision  Decision  Decision  Decision  Decision  process 
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Figure  3:  The  Assessment  Process  is  Based  on  a  Stepwise 
Decision  to  Engage  Resources  on  the  Assessment  Task. 


The  assessment  process  is  initiated  when  a  question  is  raised  or  problem  encountered  with  the  information  to 
answer  lacking,  which  may  be  coming  from  an  intern  or  external  source.  One  assumption  in  this  guide  is  that  the 
commander  has  the  final  say  on  how  resources  are  used.  Consequently,  even  if  it  is  not  the  commander  who  is 
the  “owner  of  the  problem”  the  process  is  designed  around  a  series  of  successive  decisions  made  by  the 
commander.  In  practice,  it  might  be  necessary  to  get  approval  from  other  stakeholders  -  “problem  owners”  - 
as  well.  For  the  purpose  of  this  guide,  the  term  commander  is  used  to  indicate  the  individual  who  has  the 
mandate  to  make  the  adequate  decisions  to  bring  the  process  forward. 

The  decision  points  included  in  the  assessment  process  give  the  commander  the  opportunity  to  adjust  or  even 
halt  the  assessment  task  in  case  resources  are  needed  elsewhere.  Consequently,  the  process  is  deliberately 
designed  to  minimize  the  use  of  resources  for  each  step.  A  small  team  has  a  “first  look”  at  the  problem  and 
suggest  a  way  to  address  it  before  a  more  detailed  planning  is  executed.  If  a  problem  which  initially  seemed 
manageable  turns  out  to  be  to  resource  demanding  for  the  staff  to  handle,  the  assessment  task  should  be  halted  as 
early  as  possible  and,  if  the  problem  is  significant,  external  resources  requested. 

The  Guide  gives  substantial  attention  to  deepening  the  understanding  of  what  really  the  question  or  problem  is. 
This  helps  to  achieve  a  good  cost-benefit  trade-off  and  optimal  use  of  resources.  The  steps  after  the  conclusions 
are  drawn  and  the  answer  to  the  question  is  given,  e.g.  implementing  the  new  tool  or  new  way  of  working  are 
beyond  the  scope  of  this  Guide. 
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Figure  4:  Guide  for  the  Assessment  Process  in  Five  Steps  and  Twenty-Two  Leading  Questions. 
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2.0  GUIDE 

2.1  Step  1:  Framing  the  Question 

The  trigger  for  setting  up  an  assessment  process  often  comes  from  an  emerging  problem,  a  new  tool, 
new  conditions,  thus  in  general  an  uncertainty  about  something  that  results  in  a  question  (e.g.  why,  what,  how). 
Such  question  can  come  from  inside  or  outside  the  organisation  unit.  It  is  assumed  that  the  answer  and 
information  for  the  answer  is  not  readily  available  and  requires  a  more  elaborate  analysis  and  experimentation, 
i.e.  trying  out. 

Framing  the  question  or  problem  is  intended  to  get  to  what  the  problem  really  is  about.  This  will  ascertain  to  find 
an  answer  to  the  right  question.  Also  this  will  help  to  use  the  resources  for  the  assessment  as  efficient  as  possible. 
Exploring  the  fundamental  aspects  of  the  problem  and  available  resources  is  explored  and  clarified  in  the  format 
of  a  dialogue.  The  dialogue  addresses  the  questions  presented  in  the  sub-sections  below. 

2.1.1  Ql-1:  What  are  the  main  features  of  the  problem? 

The  question  or  expressed  need  must  be  developed  into  an  explicit  formulated  problem  statement  -  why  it  is  a 
problem,  covering  the  basic  characteristics  of  the  problem.  In  this  context  three  major  problem  types  can  be 
distinguished: 

•  A  conditions  or  input  problem.  Problems  with  available  structures  or  resources  -  people,  materiel, 
finances,  procedures,  e.g.  are  available  resources  insufficient  or  inadequate?,  or  the  procedures  are  not 
aligned  with  the  (new)  setup? 

•  Problems  with  how  available  resources  are  transformed  and  used  -  process  problem.  For  example, 
although  adequate  resources,  are  the  processes  in  which  the  resources  are  used  insufficient  or  inadequate? 

•  Problems  with  performance  -  output  problem  -  for  example,  although  having  adequate  resources  and 
processes  according  to  best  practices,  is  performance  insufficient  or  inadequate? 

2.1.2  Ql-2:  Which  resources  are  available  for  planning  and  executing  the  assessment? 

Resources  for  carrying  out  the  planning  task  must  be  clearly  defined.  The  following,  non-exhaustive, 
considerations  could  be  clarified: 

•  What  are  the  time  constraints? 

•  What  kind  of  knowledge/expertise  is  necessary  to  include  in  the  planning  team? 

•  Are  there  any  specific  needs  of  equipment,  facilities  or  technology? 

The  section  is  concluded  with  a  decision  by  the  commander,  as  direct  or  delegated  problem  owner,  whether  to 
make  an  initial  analysis  of  the  problem.  The  decision  should  include  a  clearly  defined  “commander’s  problem 
statement”  -  roughly  corresponding  to  “commander’s  intent”.  The  decision  should  also  include  of  which 
resources  can  be  used  for  the  next  step  -  initial  problem  analysis. 

2.2  Step  2:  Initial  Problem  Analysis 

The  person  tasked  for  the  assessment  may  need  to  build  an  assessment  team  in  order  to  be  able  to  perform  the 
initial  analysis  of  the  assessment  task.  The  intent  is  to  have  a  first  look,  without  digging  too  much  into  details, 
on  what  it  takes  to  complete  the  assessment  and  what  the  potential  gains  are. 
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The  result  of  this  work  is  presented  in  an  “initial  planning  brief’.  The  content  of  the  brief  should  give  an 
overview  of  what  it  takes  to  execute  an  assessment.  The  aim  is  to  give  an  overview  with  enough  information  to 
make  it  possible  for  the  commander  to  decide  whether  the  assessment  is  worth  the  effort  -  primarily  whether  it  is 
worth  initiating  a  more  detailed  planning.  Based  on  the  information  given  at  the  brief,  the  commander  decides  on 
how  to  proceed  and  with  which  resources. 

The  first  effort  in  the  development  of  the  assessment  is  to  define  what  really  the  problem  behind  the  question  is 
and  which  stakeholders  are  related  to  this.  Initial  formulations  of  a  problem  statement  often  come  with  an 
implicit  analysis  or  solution.  This  step  is  to  identify  the  problem  behind  the  stated  problem  to  avoid  potential 
biases. 

The  initial  problem  analysis  should  address  the  questions  presented  in  the  sub-sections  below. 

2.2.1  Q2-1:  What  do  you  really  want  to  know? 

This  is  about  reviewing  and  developing  the  “commander’s  problem  statement”  developed  in  dialogue  with  the 
commander.  The  primary  aim  is  to  deepen  the  initial  analysis  with  a  broader  set  of  people  and  expertise  in  the 
team.  By  taking  a  systems  perspective  (Figure  5)  it  is  possible  to  precisely  define: 

•  What  system  to  study  (that  is  the  integral  setting  of  Inputs,  Processes,  and  Output  in  Context)? 

•  What  outputs  is  the  system  set  up  for  to  realise? 

•  What  effects  are  assumed  to  result  from  the  system’s  outputs? 


f 

\ 

ENVIRONMENT-  enablers/  obstacles/  stakes/  stakeholders 

— 

_ y 

INPUTS 

(conditions  and  provisions 
enabling  the  processes  to  run) 

e.g. 

Tasking 

Mandates 

Resources 

Means 

Structures 

Procedures 


PROCESSES 

(activities  required  to  realise  the 
defined  outputs) 

e.g. 

Gathering  information 
Communicating 
Deciding 
Planning 

Monitoring  /  Controlling 
Evaluating 


(intermediate) 
OUTPUTS 

(results,  products  that 
contribute  to  achieve  the 
objectives) 

e.g. 

Summative  conclusion 
Validated  instruments 
Improved  performance 
Operational  readiness 

v _ y 


OBJECTIVES 

(higher  level 
goals  -  added 
value  for 
stakeholders) 


EFFECTS 

(short  term 
and  longer 
term  effects) 


Figure  5:  A  System  Perspective  on  Defining  the  Problem,  with  Example  Items  to  Address. 


Based  on  the  commanders  problem  statement,  potential  relations  between  input,  process  and  output  factors 
should  be  outlined.  For  example,  if  there  is  a  problem  with  available  resources  (input),  specify  some  possible 
explanations  how  this  has  an  effect  on  processes  and  output2.  See  also  Ref.  [7]  for  an  explanation  on  the  use  of  a 
systems  perspective. 


Note  that  the  system  perspective  is  not  the  same  as  “organisational  processes”.  Instead  it  is  about  outlining  how  the  core  problem 
as  defined  by  the  answer  to  question  Q 1  - 1 :  “What  are  the  main  features  of  the  problem?”  might  be  influenced  and  is  influencing 
other  important  factors. 
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2.2.2  Q2-2:  What  statement  or  conclusion  do  you  need  to  give? 

This  is  about  finding  out  what  statements  or  conclusions  are  needed  or  expected  in  the  communication  about  the 
results  -  when  is  the  result  considered  to  be  convincing,  a  criterion  for  follow-on  decisions.  For  instance,  if  a 
substantial  improvement  of  the  throughput  of  a  process  is  being  sought,  a  large  time  gain  may  be  the  objective. 
Also,  if  the  result  of  the  assessment  is  a  source  for  decisions  on,  e.g.  a  substantial  commitment,  then  extra 
considerations  may  be  advised  in  setting  up  and  measuring  the  effects.  In  other  words,  this  step  is  about 
verifying  that  the  assessment  set-up  addresses  the  perceived  problem  to  a  desired  criterion  level. 

2.2.3  Q2-3:  Where  do  you  get  the  information  to  make  a  statement  about  the  system? 

This  is  about  defining  a  setting  in  which  it  is  possible  gather  information  about  the  assessment.  The  setting 
should  be  defined  in  four  elements: 

•  Context.  The  operational  setting  and  the  scenario  in  which  the  system  performs. 

•  Actors.  For  example,  the  units,  opponents,  stakeholders  involved. 

•  Environment.  For  example,  weather,  ground  conditions 

•  Content.  For  example,  the  critical  events  in  the  scenario. 

Concerning  the  operational  setting,  experimentation  in  practice  are  being  done  in  diverse  context,  but  rarely  in 
real  operations.  The  possibility  to  experiment  with  new  procedures  or  new  tools  can  in,  e.g.  a  Command  Post 
Exercise  (CPX)  or  Computer  Assisted  Exercise  (CAX).  All  depends  on  how  much  time  and  manpower  the 
commander  sees  as  adequate  to  include  in  the  exercise.  This  may  also  count  for  Live  Exercise  (L1VEX)  although 
there  time-space  relations  critically  limit  experimentation.  If  the  seriousness  of  the  question  or  problem  requires 
a  high  level  of  control,  such  as,  e.g.  different  groups  working  in  parallel,  than  a  specific  setup  independent  of 
other  stakes  may  be  necessary. 

2.2.4  Q2-4:  What  is  the  best  way  to  measure  the  factors  included  in  the  problem? 

One  can  say,  that  the  stronger  the  results  statement  needs  to  be,  the  stronger  the  measurement  methods  need  to 
be.  In  general  the  combination  of  multiple  methods  makes  the  assessment  stronger.  Measurement  methods  that 
can  be  used  during  field  studies  are  observations,  questionnaires,  and  objective  measures.  Table  1  summarizes 
the  general  characteristics  of  the  different  measurement  methods  regarding  different  factors.  Specific  useful 
methods  that  can  be  found  in  the  literature  can  be  added  to  these  general  methods.  For  instance,  communications 
analysis  can  be  used  to  analyse  the  processes  of  interaction  as  part  of  collecting  Observations  data  (e.g.  counting 
the  turn  taking),  or  as  an  objective  measure,  e.g.  using  time  dimension  of  communications. 
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Table  1:  Measurement  Methods. 


Observation 

Questionnaire  /  Interview 

Objective  Measures 

Input:  Resources 
and  Restrictions 

Observers  assess  level  or 
characteristics  of  resources 
and  conditions  for  process 

Ask  participants  how  the 
perceived  available 
resources  and  input 
conditions 

Log  and  count  resources 
and  input  conditions 

Process: 

Behaviour 

Observers  assess  quality  of 
Process  and  score 
behaviour 

Ask  participants  to  assess 
the  process  and  how  they  or 
others  behaved 

Log  process  and  count  if 
certain  processes  took 
place,  and  how  much  they 
were  shown 

Output: 

Performance 

Observers  assess  quality  of 
performance 

Ask  participants  to  assess 
their  own  or  others’ 
performance 

Give  objective  score  on  task 

Outline  some  options  for  how  to  use  different  measurement  methods  to  study  the  problem  on  basis  of: 

•  Does  it  measure  the  outcome  you  want  it  to  measure? 

•  Can  it  be  incorporated  in  the  experimentation  setup? 

The  answers  to  the  questions  of  the  “initial  problem  analysis”  are  summarized  and  presented  in  a  brief  to  the 
commander  including  the  following  items: 

•  Defined  Problem  statement  with  comments  from  the  planning  team. 

•  Background  to  the  perceived  problem. 

•  Rationale  (if  applicable):  Why  this  is  an  important  problem? 

•  Goal  (if  applicable):  What  is  the  expected  “end  state”  of  an  assessment? 

•  Time  (if  applicable):  When  is  an  assessment  of  the  problem  necessary  to  be  completed? 

•  Suggested  design  options: 

•  What  do  we  need  to  know  in  order  to  resolve  the  problem? 

•  How  can  we  get  this  information? 

•  Preferably  two  to  three  options  with  a  preferred  choice? 

•  For  each  option:  Which  resources  are  necessary  for  the  assessment.  Is  the  assessment  task  possible  to 
solve  with  available  resources  or  is  it  necessary  to  include  external  support? 

As  in  any  other  planning  task  the  commander  has  to  make  the  decision  on  whether  to  proceed.  The  decisions 
concerns  to  give  the  order  to  the  assessment  team  to  execute  the  planning  with  defined  resources  and  time 
constraints  in  order  to  assess  a  problem  specified  in  the  “commanders  problem  statement”. 

2.3  Step  3:  Detailed  Assessment  Planning 

The  assessment  team,  possibly  augmented  with  additional  resources,  then  makes  an  in  depth  planning  of  the 
assessment  task.  The  work  of  preparing  the  assessment  is  important  as  it  determines  the  possibility  success. 
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The  objective  is  that  the  question,  the  measures  and  assessment  design  are  specified  in  a  plan:  Specific, 
Measurable,  Achievable/Agreed,  Realistic,  Time-bound  (SMART).  The  development  of  the  detailed  assessment 
plan  includes  definitions  of  indicators  (what  is  being  measured)  and  corresponding  procedures  to  collect 
infonnation  about  these  indicators. 

The  result  from  the  assessment  planning  is  presented  in  an  “assessment  brief’.  Based  on  the  information  given  at 
the  brief,  the  commander  decides  on  whether  to  proceed  or  not  with  the  assessment. 

The  preparation  phase  should  address  the  questions  presented  in  the  sub-sections  below. 

2.3.1  Q3-1:  What  do  I  need  to  look  for? 

This  is  about  outlining  the  details  of  the  statement  or  conclusion  you  want  to  give  defined  in  question  Q2-2. 
For  each  identified  factor  or  system  characteristic  a  number  of  measures  should  be  identified,  e.g.  a  concrete 
formulation  of  what  to  assess.  The  following  sub  questions  might  serve  as  a  guideline: 

•  What  are  the  important  characteristics  of  the  input  factors? 

•  In  which  way  might  they  vary? 

•  What  different  forms  may  the  implementation  of  the  processes  take? 

•  What  other  factors  might  be  of  relevance? 

Multiple  measures  for  each  factor  may  be  defined.  The  principle  is  illustrated  in  Table  2. 

Table  2:  Example  of  a  Table  Showing  the  Connection  Between  Factors  and  Measures. 


Factor 

Measure 

Output  factor  FI 

Candidate  measure  Fl.l 

Candidate  measure  FI. 2 

Output  factor  F2 

Candidate  measure  F2.1 

Process  factor  F3 

Candidate  measure  F3.1 

Candidate  measure  F3.1 

2.3.2  Q3-2:  In  which  specific  setting  are  you  going  to  collect  the  data? 

This  is  about  designing  different  options  for  carrying  out  the  assessment,  e.g.  the  outlining  the  details  of  the 
context  in  which  data  will  be  collected.  The  following  sub  questions  might  serve  as  a  guideline. 

•  What  activities  are  in  the  focus  of  your  assessment? 

•  What  scenario  matches  best  the  activities  under  assessment? 

•  When  and  where  are  you  going  to  collect  data? 

•  What  is  the  best  setting  for  the  assessment? 

It  is  recommended  that  a  few  representative  activities  are  selected  in  order  to  focus  the  measurement  and  manage 
the  resource  costs  of  the  measurement.  Selecting  such  activities  may  be  self-evident  after  the  previous  questions 
have  been  completed. 
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A  general  consideration  is  whether  to  use  the  ‘natural’  day-to-day  business  as  a  setting  for  the  assessment,  or  if  it 
is  necessary  to  create  a  specific  setting  for  assessing.  It  might  be  necessary  to  insert  events  or  stimuli  that  trigger 
the  behaviours.  Mostly,  a  natural  setting  may  be  appropriate  and  authentic,  but  can  be  problematic  because  of 
operational  risks,  interfering  with  actual  processes,  limited  opportunity  of  measurement,  the  focal  activity  occurs 
rarely. 

The  elements  of  the  focal  activities  that  are  going  to  be  measured  are  defined  and  represented  in  a 
timeline/process  map  for  the  focal  activity  with  an  indication  of  when  and  where  the  measures  will  be  taken. 

2.3.3  Q3-3:  What  data  collection  instruments  are  needed? 

This  is  about  defining  the  details  about  how  to  gather  detailed  information  about  the  factors  and  measures. 
The  following  sub-questions  might  serve  as  a  guideline. 

•  How  does  each  factor  and  measure  present  itself  in  the  organization? 

•  What  measurement  methods  are  appropriate? 

•  What  is  the  specific  measurement  task?  For  example:  which  specific  questions  to  ask  in  a  questionnaire? 

Ideally,  multiple  measurement  methods  should  be  defined  to  gather  information  concerning  a  single  candidate 
measure.  The  principle  is  illustrated  in  Table  3. 


Table  3:  Example  of  a  Table  Showing  a  Summary  of  Measurement 
Methods  and  Measures  for  the  Different  Factors. 


Factor 

Measure 

Measurement  Method 

Output  factor  FI 

Candidate  measure  Fl.l 

Observation  Fl.l 

Questionnaire  Fl.l 

Interview  Fl.l 

Candidate  measure  FI. 2 

Questionnaire  FI. 2 

Observation  FI. 2 

Candidate  measure  FI. 3 

Questionnaire  FI. 3 

Interview  FI. 3 

Process  factor  F2 

Candidate  measure  F2.1 

Observation  F2. 1 

Log  F2.1 

Candidate  measure  F2.2 

Observation  F2.2 

2.3.4  Q3-4:  How  complete  is  the  proposed  assessment  plan? 

This  is  about  creating  a  feasible  assessment  plan,  which  meets  time,  resources,  and  cost  constraints. 
The  following  sub  questions  might  serve  as  a  guideline.  Sample,  non-exhaustive,  questions  are: 

•  What  is  the  budget  to  support  the  assessment? 

•  How  many  people  will  be  on  the  core  assessment  team? 
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•  Which  expertise  is  necessary  to  include  in  the  team? 

•  Are  there  any  time  constraints  regarding  data  collection? 

•  Are  there  any  time  constraints  regarding  analysing  and  interpret  data? 

•  Are  there  any  time  constraints  to  make  a  final  report  or  briefing? 

•  Are  there  any  the  risks  and  opportunities  associated  with  the  assessment,  what  actions  you  can  take 

when  these  occur,  and  what  resources  might  be  consumed  or  saved  by  them? 

Based  on  these  considerations  a  more  detailed  assessment  plan  can  be  outlined.  This  means  to  decide  on  the 
scope  of  the  assessment  for  each  selected  measure.  How  many  participants  will  be  involved  in  the  assessment? 
How  many  times  will  the  focal  activities  be  observed?  How  much  data  will  be  collected?  Note  that  it  is  rarely 
necessary  to  do  a  comprehensive  survey  or  observation,  such  as  surveying  everyone  in  the  brigade  or  observing 
every  instance  of  a  focal  activity.  In  most  cases,  a  smaller  typical  sample  will  provide  a  good  estimate.  On  the 
other  hand,  to  set  up  a  naturalistic  situation  a  diversity  of  participants  may  be  needed  even  from  outside  the 
sourcing  unit. 

Analogical  to  a  tactical  plan,  the  assessment  plan  is  a  written  script  that  describes  in  detail  what  has  to  be  done, 
at  what  time,  and  who  is  responsible  for  the  execution.  The  format  of  the  assessment  plan  will  vary  between 
assessment  tasks.  Irrespective  of  format,  it  should  minimally  include  an  answer  to  the  following  aspects: 

•  Scenario  setting: 

•  The  content  and  timing  of  the  scenario  and  events. 

•  Necessary  preparations  for  the  scenario  and  events. 

•  Characteristics  of  participants. 

•  How  to  motivate  participants  to  engage  in  the  assessment. 

•  The  information  briefing  and  framing  of  the  mind-set  of  the  participants. 

•  Instruction  and  debriefing  of  participants. 

•  Measurement: 

•  Data  collection  time  line. 

•  How  different  kind  of  instrument  of  measurement  will  be  used. 

•  Rotation  schedule,  including  time  needed  for  different  parts  of  the  assessment. 

•  How  to  secure  data. 

Subsequently,  the  answers  to  the  questions  in  the  preparations  phase  are  summarized  and  presented  in  a  brief  to 
the  commander  including  the  following  items: 

•  Recapitulation  of  the  perceived  problem. 

•  Recapitulation  of  the  rationale:  Why  this  is  an  important  problem? 

•  Recapitulation  of  the  goal:  What  is  the  expected  “end  state”  of  an  assessment? 


14 


STO-TR-HFM-211 


AN  APPLIED-SCIENTIFIC 
APPROACH  TO  FIELD  ASSESSMENTS - 
TRY  IT  OUT”  -  A  FIELD  GUIDE  FOR  PRACTITIONERS 


(( 


•  Assessment  design:  How  to  execute  the  assessment: 

•  What  do  we  need  to  know  in  order  to  resolve  the  problem  (which  factors  to  collect  data  on)? 

•  How  do  we  get  this  information? 

•  Timeline  of  assessment. 

•  Which  resources  are  necessary  for  the  assessment? 

The  written  assessment  plan,  together  with  ready-to-use  versions  of  data  collection  instruments,  questionnaires, 
observer  protocols,  etc.,  are  enclosed  with  the  briefing  material. 

2.4  Step  4:  Preparation  and  Execution 

During  the  Prepare  and  Execute  phase,  the  data  collection  plan  is  carried  out.  The  assessment  team  manages  the 
preparation  and  execution  of  the  assessment,  i.e.  collection  of  information/data  about  the  identified  indicators, 
with  their  assigned  resources.  Actions  are  undertaken  in  order  to  coordinate  efforts  and  to  assure  quality  of 
collected  information.  This  step  results  in  a  compiled  set  of  data  with  information  about  the  stated  problem. 
The  results  are  presented  in  a  first  impression  brief. 

The  execution  phase  is  divided  into  three  sub-phases: 

•  Before  the  actual  assessment. 

•  During  the  execution  of  the  assessment. 

•  Directly  after  the  assessment  is  completed. 

During  these  sub  phases,  a  number  of  questions  should  continuously  be  addressed. 

2.4.1  Sub-Phase:  Before  the  Actual  Assessment 

This  is  about  the  final  checks  before  the  assessment  is  initiated,  performing  dry  runs  to  verify  if  all  technical 
aspects  of  the  assessment. 

2. 4. 1.1  Q4-1:  Will  the  assessment  run  as  intended? 

This  is  about  ensuring  that  the  assessment  plan  is  adequately  implemented.  Before  of  the  assessment,  different 
parts  of  the  study  (i.e.  the  system,  measurement  methods,  instruction)  should  be  tested  to  assess  whether  they 
work  like  they  are  supposed  to  in  the  time  scheduled  for  it.  Consequently,  a  trial  run  of  the  assessment  is 
desirable  as  it  allows  for  flaws  in  the  system  or  in  the  scenario  to  be  discovered  before  the  actual  study. 

Ideally,  such  trial  runs  can  be  executed  already  during  the  detailed  planning.  However,  in  many  cases  necessary 
resources  might  not  be  available  during  planning.  For  example,  units  participating  in  the  assessment  might  have 
other  tasks  to  solve.  Still,  some  parts  of  the  assessment  process  can  be  tested  early  without  excessive  resource 
investment.  For  example,  questionnaires  might  be  tested  on  representative  individuals. 

2. 4. 1.2  Q4-2:  Have  any  threats  to  assessment  quality  emerged  since  the  assessment  plan  was  outlined? 

When  the  assessment  has  been  initiated,  time  will  most  likely  be  a  limiting  factor  for  success  of  the  execution. 
Failure  in  equipment  or  misunderstandings  in  the  schedule  could  have  a  devastating  effect.  Moltke’s  famous 
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statement  that  “No  campaign  plan  survives  first  contact  with  the  enemy”  (originally  in  Ref.  [9],  found  in 
Ref  [10])  is  true  for  the  assessment  task  as  well. 

The  use  of  checklists  to  review  the  assessment  design  setting  on  a  regular  basis  is  recommended.  The  basic 
content  of  such  checklists  are: 

•  Equipment  -  all  equipment  necessary  for  the  assessment.  Experience  shows  that  an  early  check  of 
functions  in  systems  every  morning  during  the  execution  has  spared  much  of  the  assessment  problems. 

•  Schedule  and  logistics  -  are  there  are  any  frictions  (i.e.  meals,  security  clearance,  transports)  that  can  be 
solved  in  advance. 

•  Personnel  -  if  there  is  any  deviation  in  the  planned  manning.  This  must  be  noted  so  that  it  can  be 
included  in  the  analysis.  The  simplest  way  of  doing  this  is  to  create  a  template  with  observed  deviation, 
what  task  this  role  had,  how  the  shortage  was  handled  and  the  estimated  effect  of  this  deviation. 

Deviations  should  be  noted,  in  order  to  be  able  to  include  these  in  the  analyses. 

Daily  assessment  team  meetings  in  order  to  Synchronize  actions  in  advance  of  assessment  activities  aims  to 
make  sure  that  everyone  knows  what  to  do,  how  to  do  it,  and  how  the  assessment  team’s  activities  should  be 
coordinated.  In  these  meetings,  the  schedule  for  the  day(s)  should  be  reviewed  and  in  order  to  adjust  instructions 
for  data  collection,  and  discuss  any  other  coordination  needs. 

2.4.2  Sub-Phase:  During  the  Execution  of  the  Assessment 

This  is  about  monitoring  the  assessment  process  and  managing  unexpected  events  in  the  light  of  the  assessment 
objectives  and  plan.  Especially  with  a  large  assessment  team,  collecting  data  in  a  limited  amount  of  time,  within 
a  setting  where  regular  updates  on  the  assessment  are  expected  a  Daily  Assessment  Team  Meeting  (DATM)  is 
required. 

The  purpose  of  these  DATMs  is  to  review  the  assessment  activities  and  data  gathered  and  the  progress  of  the 
assessment  and  allocate  actions  when  adjustment  is  needed.  Ideally,  the  commander  or  an  individual  assigned  by 
the  commander  takes  part  in  the  meeting.  The  DATM  should  address  a  number  of  questions  related  to  data 
collection,  assessment  obstacles,  additional  resources  or  help,  indicative  intermediate  assessment  results. 
The  DATM  concludes  with  a  preview  of  events  and  actions  the  following  day. 

2.4.2. 1  Q4-3:  Has  the  data  been  collected  and  secured  according  to  plan? 

This  is  about  reviewing  the  data  collection  plan  and  check  if  all  data  has  been  obtained.  Has  the  assessment  team 
access  to  obtained  data  or  are  there  any  additional  actions  needed  to  secure  data? 

2. 4. 2. 2  Q4-4:  Is  there  any  additional  need ,  compared  to  the  assessment  plan,  for  coordination  of  the 
assessment? 

This  is  about  adjusting  the  plan  for  upcoming  data  collection  efforts  in  order  to  address  any  identified  but 
unforeseen  changes  in  circumstances. 

2. 4. 2. 3  Q4-5:  Were  there  any  unexpected  events  that  might  influence  the  data  and  results? 

This  is  about  deciding  whether  the  circumstance  under  which  data  was  collected  was  adequate  to  arrive  at  a  solid 
conclusion  on  the  assessment  question. 
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The  notes  or  decisions  of  the  DATM  are  documented  in  a  Daily  Activity  Note.  Deviations  from  the  plan  that 
may  influence  the  data  should  be  documented  and  taken  into  consideration  during  subsequent  analyses. 
The  simplest  way  of  doing  this  is  to  create  a  template  with  the  columns: 

•  Observed  event  and  deviation. 

•  When  did  it  occur? 

•  Who  is  responsible  for  handling? 

•  How  was  it  handled? 

•  What  is  the  estimated  effect  of  this  deviation? 

2.4.3  Sub-Phase:  Immediately  After  the  Assessment  is  Completed 

This  is  about  the  initial  analysis  of  obtained  data,  with  first  impressions. 

2. 4. 3.1  Q4-6:  What  are  the  first  impressions  from  the  assessment? 

At  the  end  of  the  assessment  session  often  a  so  called  ‘hot  wash-up’  is  organised  for  immediate  feedback  of  first 
impressions  to  the  participants.  In  general,  after  such  assessment  session  participants,  and  in  particular  those 
from  outside  the  unit,  will  distribute  and  involved  in  other  activities,  so  that  this  may  be  the  only  moment  to  get 
some  idea  about  the  results.  Input  for  such  briefing  comes  from  the  daily  observations.  Since  a  First  Impression 
generally  lacks  serious  analysis  of  the  data  gathered,  it  is  best  is  to  stay  away  from  interpretations  or  meaning 
and  stick  to  the  factual  observations.  The  same  counts  for  discussing  preliminary  results  with  stakeholders 
directly  after  the  execution.  If  a  First  Impression  Report  is  required  one  could  focus  on: 

•  The  purpose  and  the  objectives  of  the  assessment. 

•  How  what  data  were  collected  and  how  the  collection  process  went. 

•  Presenting  descriptive  data  (e.g.  how  many  people  filled  out  the  questionnaire). 

•  Presenting  preliminary  results,  factual  observations  or  data  (not  conclusions!),  consisting  of  simple 
aggregations  of  raw  data. 

•  Describing  the  further  analysis  process  and  projected  delivery  of  the  full  report. 

Presenting  preliminary  results  may  have  an  opportunity  to  get  feedback  from  the  participants,  which  may 
support  the  analysis  and  interpretation  of  the  data.  For  example,  participants  may  be  able  to  give  there  state  of 
mind  or  contextual  background  related  to  particular  events  and  decisions  or  actions  they  performed  during  the 
session. 

2.5  Step  5:  Analysis  and  Reporting 

After  the  completed  assessment,  the  data  collected  during  the  execution  are  analysed,  interpreted,  and  reported 
by  the  team.  The  approach  to  do  this  is  highly  dependent  on  the  specific  contextual  circumstances  and  should  be 
defined  in  the  assessment  plan.  The  main  result  should  the  answer  on  the  stated  problem. 

In  addition,  the  team  should  also  make  an  assessment  on  the  quality  of  the  obtained  data.  The  result  is  presented 
in  a  “results  brief’.  Based  on  the  information  given  at  the  brief,  the  commander  may  decide  on  any  further 
action. 
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The  analysis  and  report  phase  is  divided  into  three  sub  phases: 

•  Analyse  and  interpret; 

•  Review  findings;  and 

•  Reporting. 

2.5.1  Sub-Phase:  Analyse  and  Interpret 

This  is  about  condensing  and  characterising  the  obtained  data  from  the  assessment  and  subsequently  considering 
the  implications  from  these  findings. 

2. 5.1. 1  Q5-1:  How  can  the  results  be  characterized? 

The  data  may  initially  be  in  written  form,  on  questionnaires,  rating  sheets,  or  various  other  formats.  It  is  helpful 
to  gather  them  up  onto  spreadsheets  in  order  to  do  adequate  analyses,  e.g.  calculations  of  quantitative  data  such 
as  averages  or  qualitative  sum-ups  or  content  analysis  of  qualitative  data.  The  aim  is  to  characterize  the  outcome 
based  on  the  collected  data. 

Table  4  illustrates  the  principle  for  how  the  outcome  from  the  collected  data  should  be  connected  to  the  different 
factors.  Each  set  of  data  is  represented  by  a  summary  of  the  findings.  Depending  on  the  nature  of  the  collected 
data,  this  summary  might  vary  but  it  is  typically  a  mean  value,  a  summary  of  comments,  most  selected  items,  etc. 


Table  4:  Example  of  a  Table  Showing  How  Outcomes  Could  be  Connected  to  the  Different  Factors. 


Factor 

Measure 

Measurement 

Method 

Summary  of  Obtained 

Results 

Output  factor  FI 

Candidate  measure  Fl.l 

Observation  Fl.l 

A  majority  of  decisions  was 
regarding  . . . 

Questionnaire  Fl.l 

Subjects  generally  preferred 

Interview  Fl.l 

Subjects  mentioned  . . . 

Candidate  measure  FI. 2 

Questionnaire  FI. 2 

Mv  4,1  and  3,7 

Observation  FI. 2 

Two  alternatives  were  used  . . . 

Candidate  measure  FI. 3 

Questionnaire  FI. 3 

The  options  were  ranked  in  the 
following  order  . . . 

Interview  FI. 3 

Subjects  were  positive 
regarding  . . . 

Process  factor  F2 

Candidate  measure  F2.1 

Observation  F2. 1 

System  was  mainly  used  by. . . 

Fog  F2.1 

The  system  was  accessed  . . . 

Candidate  measure  F2.2 

Observation  F2.2 

No  observed  deviations  from  ... 
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2. 5. 1.2  Q5-2:  What  do  the  findings  mean? 

Subsequently,  results  should  be  qualified  in  terms  of  what  these  results  mean?  Is  there  a  change  or  a  difference? 
Three  basic  approaches  can  be  used  answer  this  question.  Which  method  is  feasible  depends  on  the  design  of  the 
assessment. 

•  Comparing  results  with  a  norm  or  criterion. 

•  Comparing  results  over  time  for  each  unit  including  a  control  unit  with  no  intervention. 

•  Comparing  the  results  between  units  being  each  other’s  benchmark. 

Consequently,  this  is  about  deciding  whether  the  outcome  was  according  to  what  was  expected  or  not.  For 
example,  did  the  limitations  in  the  input  factor  affect  process  and  output  factors  as  expected?  Was  the  process 
executed  according  to  standards?  Was  performance  in  line  with  the  expected? 

Table  5  illustrates  the  principle  on  how  this  interpretation  should  be  connected  obtained  results  for  the  different 
factors. 


Table  5:  Example  of  a  Table  Showing  the  Assessment  How  What 
Different  Obtained  Results  Indicate  in  Terms  of  Performance. 


Factor 

Measure 

Measurement 

Method 

Summary  of 

Obtained  Results 

Indicates 

Output 
factor  FI 

Candidate 
measure  Fl.l 

Observation  Fl.l 

A  majority  of  decisions 
was  regarding ... 

Better  performance 

Questionnaire  Fl.l 

Subjects  generally 
preferred . . . 

Lower  performance 

Interview  Fl.l 

Subjects  mentioned  . . . 

Not  conclusive 

Candidate 
measure  FI. 2 

Questionnaire  FI. 2 

Mv  4,1  and  3,7 

Better  performance 
regarding  . . . 

Observation  FI. 2 

Two  alternatives  were 
used  . . . 

Better  performance 

Candidate 
measure  FI. 3 

Questionnaire  FI. 3 

The  options  were 
ranked  in  the  following 
order  . . . 

Lower  performance 

Interview  FI. 3 

Subjects  were  positive 
regarding  . . . 

Better  performance 

Process 
factor  F2 

Candidate 
measure  F2.1 

Observation  F2. 1 

System  was  mainly 
used  by  . . . 

Process  according  to 
expected  . . . 

Log  F2.1 

The  system  was 
accessed  . . . 

Process  deviated  . . . 

Candidate 
measure  F2.2 

Observation  F2.2 

No  observed 
deviations  from  . . . 

Not  conclusive 
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Subsequently,  the  picture  given  from  the  most  “important”  indicators  are  summarized.  What  does  the  findings 
say  about  the  “commander’s  problem  statement”? 

It  is  also  recommended  that  indicators  pointing  in  another  direction  are  mentioned  and  explained.  In  addition, 
identified  problems  in  the  assessment  should  be  commented. 

2.5.13  Q5-3:  What  do  the  results  tell  you? 

This  is  about  analysing  the  obtained  findings  in  terms  of  whether  the  outcome  was  according  to  what  was 
expected  or  not.  What  are  the  implications?  Are  there  any  lessons  learned?  Are  there  any  recommendations? 

There  is  no  unanimous  and  structured  way  to  come  up  with  a  conclusion.  However,  each  statement  in  the 
conclusion  should  be  supported  by  obtained  data,  commonly  accepted  knowledge,  or  “obvious”  circumstances. 
Finally,  the  conclusions  should  also  hold  some  recommendations  for  the  future. 

The  assessment  is  the  summarized  in  a  suitable  format.  Irrespective  of  format,  the  documentation  should  include 
the  following  major  components: 

•  Introduction  and  background  to  the  assessment. 

•  Methods  -  How  the  assessment  task  has  been  undertaken. 

•  Results  -  What  are  the  characteristics  of  the  obtained  results?  Was  the  outcome  according  to  what  was 
expected? 

•  Conclusions  -  For  example,  the  answer  to  the  assessment  question  and  recommendations  for  the  future. 

2.5.2  Sub-Phase:  Review  Findings 

This  is  about  quality  check  before  the  final  conclusions  and  recommendations  are  presented  to  the  commander. 

It  is  also  about  learning  from  successes  and  failures  in  the  process  to  improve  the  assessment  process  itself. 
The  team  should  address  two  important  questions  during  this  sub  phase. 

2.5.2. 1  Q5-4:  Are  there  any  flaws  in  the  conclusions  and  recommendations? 

Ideally,  an  After  Action  Review  (AAR)  or  a  so-called  ‘Hot  Wash-up’  shoidd  be  held  as  soon  as  possible 
following  completion  of  the  assessment  process.  The  basic  scope  of  the  AAR  is  to  present  and  discuss  the 
preliminary  findings  to  relevant  personnel  which  have  been  subjected  to  the  assessment.  An  important  objective 
is  to  collect  additional  data  which  might  explain  the  obtained  result.  Another  important  objective  is  to  anchor  the 
results  and  its  implications  within  the  assessed  units. 

The  AAR  is  a  structured  review  or  de-brief  process  for  analysing  what  happened,  why  it  happened,  and  how  it 
can  be  done  better,  by  the  participants  and  those  responsible  for  the  assessment  process.  It  can  either  be  formal  or 
informal  depending  on  the  size,  timeframe  and  recourses  available.  These  questions  may  be  helpful  in 
structuring  an  AAR: 
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What  was  planned/expected? 
What  actually  occurred? 
Which  results  were  obtained? 
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•  Why  did  it  happen? 

•  What  went  well  and  why? 

•  What  can  be  improved  and  how? 

•  What  are  the  important  implications  from  the  assessment? 

In  addition,  if  applicable,  a  draft  version  of  the  assessment  report  should  be  distributed  to  selected  stakeholders 
for  review  and  feedback. 

Obtained  comments  and  suggestions  are  subsequently  incorporated  in  the  final  report. 

2. 5. 2. 2  Q5-5:  Can  the  assessment  process  be  improved? 

In  parallel,  feedback  regarding  the  executed  assessment  should  also  be  collected  using  ‘improve’  or’  sustain’ 
(‘tips  and  tops’)  questions  with  the  assessment  team  and  participants  in  the  assessment. 

2.5.3  Sub-Phase:  Reporting 

This  is  about  Briefing  and  documenting  the  final  results  from  the  assessment. 

2.5.3. 1  Q5-6:  What  is  essential  to  include  in  the  final  documentation  and  briefing? 

The  evaluation  report  and  corresponding  briefing  material  should  contain  the  relevant  information  to  support  a 
decision  being  made  by  the  commander.  More  specifically,  the  Report  should  include: 

•  A  concise  statement  of  the  issue  prompting  the  assessment; 

•  The  context,  scope  and  aim  of  the  assessment; 

•  A  summary  of  the  assessment’s  conduct; 

•  A  summary  of  the  analysis; 

•  Key  findings  and  recommendations;  and 

•  Potential  risks,  confounding  factors,  and  limitations. 

In  addition,  it  might  also  be  appropriate  to  prepare  a  package  of  reference  material.  This  includes  compiled  raw 
data,  earlier  briefing  materials,  relevant  documents  etc.  Compilation  of  reference  material  is  important  as  it  will 
provide  future  assessments  with  adequate  inputs. 


3.0  CONCLUSIONS 

The  focus  of  this  Field  Assessment  Guide  is  on  experimentation  or  trying  out  in  a  natural  context  a  new  tool  or 
way  of  working  or  the  like  -  the  intervention  -  to  see  if  that  improves  the  performance.  The  main  techniques 
proposed  in  this  Guide  are: 

a)  Deliberated  thinking  about  what  one  really  wants  to  know,  understand,  or  find  out; 

b)  Detailed  planning  of  trying  out  the  effect  of  the  intervention;  and 

c)  Doing  this  in  a  step-wise,  documented  approach  enabling  others  to  critical  review  it. 
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Historically,  investments  in  military  systems  have  often  been  of  long  term  nature  and  thus  research  and  system 
development  had  a  perspective  of  decades  in  advance.  Competence  on  implemented  systems  developed  over 
time  based  on  officers  specializing  on  these  systems  and  developing  routine  procedures.  However,  the  last 
decades  have  shown  that  the  lifecycle  of  systems  tend  to  be  drastically  shortened.  Rapid  changes  in  threats, 
tactics,  tasks,  technology,  and  organization  create  a  demand  of  an  enhanced  ability  to  develop  technical  and 
organizational  solutions.  We  argue  that  constant  change  is  currently  the  normal  state  for  the  Armed  Forces. 
Consequently,  competence  cannot  just  be  based  on  routine.  The  only  way  to  adapt  to  change  is  to  try  out 
solutions  to  handle  these  changes.  Competence  on  change  will  be  a  crucial  factor  in  the  process  of  establishing 
and  maintaining  combat  ability.  “Trying  out”  should  become  a  more  integrated  and  natural  part  of  the  Defence 
Forces  every  day  activities.  This  guide  is  an  attempt  to  address  this  challenge. 

However,  ‘just  trying  out’  means  that  we  are  aware  and  accept  that  there  is  relatively  little  control  over  possible 
confounding  factors,  such  as,  e.g.  erroneously  allocating  the  observed  positive  effect  to  the  intervention  while  it 
may  come  from  just  the  attention  in  setting  up  the  trial  or  another  source  in  the  setup.  Also  the  lack  of 
independence  of  the  assessment  team  may  trick  the  team  to  seek  for  and  report  only  the  desirable  results. 
Carefully  designing  an  assessment  is  as  important  in  a  practice  setting  as  would  be  in  a  scientific  setting. 

The  discussions  in  the  Task  Group  have  shown  that  we  easily  shifted  to  scientific  methods  as  these  have  been 
tested  and  developed  and  are  well-proven  to  avoid  biases  and  mistakes  in  assessments.  Reducing  that  rigor  and 
still  maintaining  the  quality  that  is  needed  was  indeed  not  easy  to  decide  upon. 

While  this  Guide  is  based  on  our  practical  experience  it  has  not  been  tested  and  validated  in  practice.  We  suggest 
that  the  first  steps  in  this  could  be  done  in  the  defence  colleges  and  by  the  Training  and  Evaluation  sections  of 
the  staffs.  Feedback  on  the  guide  is  highly  welcomed. 
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